Updating catalog builder package version #693

Ciheim · 2026-01-06T18:59:39Z

Describe your changes

Issue ticket number and link (if applicable)

Checklist before requesting a review

I ran my code
I tried to make my code readable
I tried to comment my code
I wrote a new test, if applicable
I wrote new instructions/documentation, if applicable
I ran pytest and inspected it's output
I ran pylint and attempted to implement some of it's feedback
No print statements; all user-facing info uses logging module

ceblanton

Looks good to me. 2025.01.01 is the latest catalogbuilder version on our channel (https://anaconda.org/channels/NOAA-GFDL/packages/catalogbuilder/files)

ceblanton · 2026-01-06T19:28:14Z

It's not passing the tests but the failures don't clearly look catalogbuilder related

singhd789 · 2026-01-06T19:30:11Z

It's not passing the tests but the failures don't clearly look catalogbuilder related

I don't think it's catalog builder related either - I saw the same failure in my #679 PR but haven't taken a closer look yet

ilaflott

we need green checks!

this doesn't look associated with catalogbuilder directly, the error message is associated with a lack of the IPython, which is required by the intake_esm package, which catalogbuilder seems to require a specific version of that's not working here.

it's not impossible something about intake_esm changed from underneath catalogbuilder.

pay close attention to the trace in the logs!

https://github.com/NOAA-GFDL/fre-cli/actions/runs/20758852348/job/59608111480#step:8:169

ilaflott · 2026-01-13T19:40:49Z

actually, i'm realizing this is something to do with analysis-scripts / fre.analysis / #542 , and catalogbuilder changes just revealed the issue. consider the following:

the error/stack trace from pytest starts here. I clipped the output to the important bits, see link for all gory details:

fre/analysis/tests/test_subtools.py:52: AssertionError
----------------------------- Captured stdout call -----------------------------
Traceback (most recent call last):
  File "/tmp/tmpwu2tkcqs/python_script", line 3, in <module>
    paths = run_plugin('freanalysis_clouds', '/tmp/tmpvcmwustk/fake-catalog', '.', config=config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/analysis_scripts/plugins.py", line 140, in run_plugin
    return _plugin_object(name).run_analysis(catalog, png_dir, config, reference_catalog)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/freanalysis_clouds/__init__.py", line 97, in run_analysis
    catalog = intake.open_esm_datastore(catalog)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/__init__.py", line 82, in __getattr__
    driver = registry[attr[5:]]  # "open_..."
             ~~~~~~~~^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/source/__init__.py", line 30, in __getitem__
    return it.load()
           ^^^^^^^^^
...
...
...

  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/__init__.py", line 10, in <module>
    from intake_esm.core import esm_datastore
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/core.py", line 24, in <module>
    from fastprogress.fastprogress import progress_bar

...
...
...

ModuleNotFoundError: No module named 'IPython'

So, IPython is required by intake_esm, but isn't there. intake_esm is required by intake which is required by freanalysis_clouds... and this is where it gets odd! freanalysis_clouds should be getting pip install'd into it's own venv, and NOT the same environment as fre-cli / analysis-scripts.

NOAA-GFDL/analysis-scripts/user_scripts/freanalysis_clouds does require intake_esm, but again, it should be trying to use an intake_esm pip installed into a separate venv with which it is compatible

ceblanton · 2026-01-13T19:45:41Z

actually, i'm realizing this is something to do with analysis-scripts / fre.analysis / #542 , and catalogbuilder changes just revealed the issue. consider the following:

the error/stack trace from pytest starts here. I clipped the output to the important bits, see link for all gory details:
fre/analysis/tests/test_subtools.py:52: AssertionError
----------------------------- Captured stdout call -----------------------------
Traceback (most recent call last):
  File "/tmp/tmpwu2tkcqs/python_script", line 3, in <module>
    paths = run_plugin('freanalysis_clouds', '/tmp/tmpvcmwustk/fake-catalog', '.', config=config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/analysis_scripts/plugins.py", line 140, in run_plugin
    return _plugin_object(name).run_analysis(catalog, png_dir, config, reference_catalog)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/freanalysis_clouds/__init__.py", line 97, in run_analysis
    catalog = intake.open_esm_datastore(catalog)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/__init__.py", line 82, in __getattr__
    driver = registry[attr[5:]]  # "open_..."
             ~~~~~~~~^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/source/__init__.py", line 30, in __getitem__
    return it.load()
           ^^^^^^^^^
...
...
...

  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/__init__.py", line 10, in <module>
    from intake_esm.core import esm_datastore
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/core.py", line 24, in <module>
    from fastprogress.fastprogress import progress_bar

...
...
...

ModuleNotFoundError: No module named 'IPython'
So, IPython is required by intake_esm, but isn't there. intake_esm is required by intake which is required by freanalysis_clouds... and this is where it gets odd! freanalysis_clouds should be getting pip install'd into it's own venv, and NOT the same environment as fre-cli / analysis-scripts.

NOAA-GFDL/analysis-scripts/user_scripts/freanalysis_clouds does require intake_esm, but again, it should be trying to use an intake_esm pip installed into a separate venv with which it is compatible

@ilaflott thank you for getting to the bottom of this, and I'm sorry the trail leads to fre-analysis.

The existing fre-analysis stuff is being refactored, and I strongly suggest we turn off the fre-analysis tests for now.

ilaflott · 2026-01-13T19:47:59Z

I agree- @skip the fre.analysis tests... A bit sad, though

ilaflott · 2026-01-13T20:00:27Z

@Ciheim @ceblanton @singhd789

bizarre- this might actually be an intake-esm bug, catalogbuilder is having the same problem in it's pipelines that it ran successfully for recent updates. perplexing.

see https://github.com/NOAA-GFDL/CatalogBuilder/actions/runs/20970566497/job/60272504792

singhd789 · 2026-01-13T20:03:02Z

@Ciheim @ceblanton @singhd789

bizarre- this might actually be an intake-esm bug, catalogbuilder is having the same problem in it's pipelines that it ran successfully for recent updates. perplexing.

see https://github.com/NOAA-GFDL/CatalogBuilder/actions/runs/20970566497/job/60272504792

Also happened in my fre make PR: #679

Ciheim · 2026-01-13T20:07:49Z

@ilaflott Good catch. This is why we test!

ilaflott · 2026-01-13T20:24:07Z

the mystery thickens...

AnswerDotAI/fastprogress#117

ilaflott · 2026-01-13T20:30:20Z

actually, i think intake_esm's fix for this isn't up to snuff: intake/intake-esm#770

i think i officially blame intake_esm, BUT, analysis-script's failure to use an isolated environment is troubling, independently.

ilaflott · 2026-01-13T21:14:18Z

In light of exploring this, I've opened intake/intake-esm#773

@ceblanton @Ciheim @singhd789 @aradhakrishnanGFDL

codecov · 2026-01-16T14:52:20Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.30%. Comparing base (982868f) to head (21d4523).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #693   +/-   ##
=======================================
  Coverage   85.30%   85.30%           
=======================================
  Files          68       68           
  Lines        4498     4498           
=======================================
  Hits         3837     3837           
  Misses        661      661

Flag	Coverage Δ
unittests	`85.30% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Ciheim requested review from ceblanton, ilaflott and singhd789 January 6, 2026 18:59

ceblanton approved these changes Jan 6, 2026

View reviewed changes

ilaflott requested changes Jan 6, 2026

View reviewed changes

ilaflott mentioned this pull request Jan 13, 2026

Restrict fastprogress version to fix downstream ModuleNotFoundError errors intake/intake-esm#773

Open

3 tasks

Ciheim Brown and others added 2 commits January 16, 2026 09:39

changing catalog package version

1ffacd7

Update catalogbuilder version in meta.yaml

21d4523

ilaflott force-pushed the testing_catalog branch from 7e14427 to 21d4523 Compare January 16, 2026 14:39

ilaflott approved these changes Jan 16, 2026

View reviewed changes

ilaflott merged commit 07fcb72 into main Jan 16, 2026
6 checks passed

ilaflott deleted the testing_catalog branch January 16, 2026 14:57

Updating catalog builder package version #693

Updating catalog builder package version #693

Uh oh!

Conversation

Ciheim commented Jan 6, 2026

Describe your changes

Issue ticket number and link (if applicable)

Checklist before requesting a review

Uh oh!

ceblanton left a comment

Choose a reason for hiding this comment

Uh oh!

ceblanton commented Jan 6, 2026

Uh oh!

singhd789 commented Jan 6, 2026

Uh oh!

ilaflott left a comment

Choose a reason for hiding this comment

Uh oh!

ilaflott commented Jan 13, 2026

Uh oh!

ceblanton commented Jan 13, 2026

Uh oh!

ilaflott commented Jan 13, 2026

Uh oh!

ilaflott commented Jan 13, 2026

Uh oh!

singhd789 commented Jan 13, 2026

Uh oh!

Ciheim commented Jan 13, 2026

Uh oh!

ilaflott commented Jan 13, 2026

Uh oh!

ilaflott commented Jan 13, 2026

Uh oh!

ilaflott commented Jan 13, 2026

Uh oh!

codecov bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Jan 16, 2026 •

edited

Loading